Optimal Data Migration in Self-managing Storage Systems

The need for petascale storage and beyond has led to new storage architectures which are scalable, inexpensive and efficient, replacing large monolithic disk array systems. Cluster-based storage systems ([3], [5]), comprising of a large collection of small, inexpensive and unreliable storage nodes (storage bricks), are increasingly becoming popular. In such systems, data is distributed redundantly among the bricks in order to improve reliability and performance. Using a specialized data encoding scheme depending on the workload and fault tolerance requirements, can lead to substantial performance benefits as compared to a “one size fits all” model ([3]). Therefore, it is desired that these cluster-based systems provide versatility to specialize the data distribution choices for various classes of data and their workloads. For example, a file accessed sequentially should be erasure coded to reduce the amount of space needed, while one requiring lots of random accesses should be replicated. Due to the inherent complexity of distributed systems, plus the additional complexity introduced by the versatility of these systems, makes managing such systems extremely difficult. Thus, we desire such systems to be self-managing. In particular, the system should itself decide what the best data encoding for a particular storage object is. When a new storage object is introduced in the system, it can be stored using some encoding scheme. As the workload and usage characteristics for the object become known, the system should change the encoding of the object to one that achieves the best performance and the required level of fault tolerance.